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Why  GAO  Did  This  Study 

DOD  initiated  the  Study  to  address 
perceived  weaknesses  within  the 
Military  Health  System  and  to  leverage 
advances  in  civilian  business  practices. 
The  National  Defense  Authorization 
Act  for  Fiscal  Year  2015  included  a 
provision  for  DOD  to  submit  the  Study 
to  the  congressional  defense 
committees  and  for  GAO  to  review  the 
Study.  DOD  submitted  its  study  in 
February  2016.  This  report  assesses, 
among  other  things,  the  extent  to 
which  the  Study  followed  an  approach 
that  is  consistent  with  relevant 
generally  accepted  research  standards 
and  utilized  key  practices  for 
estimating  cost  savings.  GAO 
compared  the  Study  with  generally 
accepted  research  standards  that  were 
developed  by  reviewing  research 
literature  and  DOD  guidance  and  with 
key  practices  derived  from  cost¬ 
estimating  guidance. 

What  GAO  Recommends 

GAO  is  making  six  recommendations, 
including  that  DOD  conduct  a  new 
analysis  of  the  required  number  of 
active-duty  and  civilian  medical 
personnel  that  mitigates  known 
limitations;  identify  and  mitigate 
limitations  regarding  the  standard  for 
maintaining  providers’  clinical  skills; 
develop  a  strategy  for  achieving  its 
goals  for  transferring  health  care  to 
DOD  facilities  and  increasing  the 
productivity  of  active-duty  providers; 
and,  when  considering  proposed 
changes  to  facilities,  include  in  any 
accompanying  cost  estimates  an 
appropriate  level  of  detail.  DOD 
concurred  with  each  of  GAO’s 
recommendations. 
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DOD  Needs  Further  Analysis  of  the  Size, 
Readiness,  and  Efficiency  of  the  Medical  Force 


What  GAO  Found 

The  Department  of  Defense’s  (DOD)  approach  in  its  Report  on  Military  Health 
System  Modernization  (the  Study)  did  not  consistently  follow  relevant  generally 
accepted  research  standards  for  research  design  and  execution.  While  the 
Study’s  recommendations  position  DOD,  over  time,  to  take  actions  to  improve 
the  effectiveness  and  efficiency  of  the  Military  Health  System,  GAO  found  a 
number  of  shortcomings,  including  the  following: 

•  The  Study  did  not  fully  mitigate  limitations  identified  in  its  analysis  of  the 
required  number  of  active-duty  and  civilian  medical  personnel.  For  example, 
the  Study  did  not  explain  how  known  issues  with  the  military  services’ 
workforce  models  affected  the  results  of  its  requirements  analysis.  Without 
addressing  such  limitations,  DOD  will  not  have  a  full  assessment  of  its 
medical  workforce  needs. 

•  The  Study  did  not  sufficiently  identify  or  mitigate  limitations  concerning  its 
assessment  of  the  requirements  necessary  to  maintain  the  skills  of  active- 
duty  medical  providers.  For  example,  although  there  were  limitations 
concerning  the  accuracy  of  information  on  medical  providers’  workload,  the 
Study  did  not  identify  or  mitigate  these  limitations.  Having  accurate  workload 
information  is  important  to  establishing  a  sound  standard  for  maintaining  the 
clinical  skills  of  medical  providers. 

•  The  Study  established  goals  for  transferring  health  care  from  DOD’s 
purchased  care  network  into  its  own  network  of  hospitals  and  clinics  and  for 
increasing  the  productivity  of  active-duty  medical  providers,  but  did  not 
develop  a  strategy  explaining  how  these  goals  would  be  achieved.  Without 
such  a  strategy  it  remains  unclear  whether  DOD  can  achieve  its  goals  to 
transfer  health  care  from  the  purchased  care  network  into  its  own  network. 

DOD’s  estimated  cost  savings  did  not  fully  utilize  key  practices  for  developing 
such  estimates.  DOD  estimated  net  annual  savings  of  $366  million  from  changes 
to  1 0  small  hospitals  and  achievement  of  its  goals  for  recapturing  health  care 
and  increasing  the  productivity  of  active-duty  health  care  providers.  However, 
DOD  did  not  include  in  its  estimate  an  appropriate  level  of  detail  concerning  the 
calculation  of  estimated  savings,  all  potentially  significant  costs,  or  a  description 
of  the  steps  taken  by  the  Study  team  to  assess  the  reliability  of  cost  data  used  to 
develop  the  estimate.  For  example,  the  Study  recommended  that  a  number  of 
inpatient  facilities  be  closed,  but  GAO’s  analysis  found  that  the  Study  did  not 
identify  estimated  costs  associated  with  these  changes.  As  a  result,  DOD’s  cost 
savings  estimate  did  not  present  a  full  and  accurate  picture  of  possible  costs  and 
savings. 
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Figure  1:  Military  Health  System  (MHS)  Organizational  Structure 
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Congressional  Committees 

Spending  on  the  Military  Health  System  (MHS)  has  grown  substantially 
over  the  past  10  years,  increasing  from  about  $24.5  billion  in  fiscal  year 
2005  to  about  $42  billion  in  fiscal  year  2015. 1  To  address  perceived 
weaknesses  within  the  MHS  and  leverage  advances  in  civilian  business 
practices,  the  Department  of  Defense  (DOD)  initiated  the  Report  on 
Military  Health  System  Modernization  (the  Study)  in  201 3. 2  Its  two  main 
goals  were  to  achieve  cost  savings  within  the  MHS  and  increase  the 
readiness  of  the  medical  force  to  support  military  operations.  The  Study 
recommended  a  number  of  interim  goals,  metrics,  and  changes  to  MHS 
business  practices,  but  highlighted  the  need  for  further  analysis  of  these 
issues,  making  a  number  of  recommendations  along  those  lines.  It  also 
recommended  some  immediate  changes,  including  changes  in 
capabilities  to  10  of  the  24  military  hospitals  with  low  numbers  of  patients, 
such  as  closing  inpatient  facilities  in  whole  or  part  for  8  hospitals.  The 
Study  estimated  that  all  of  these  recommended  changes  could  save  $366 
million  annually.  The  Under  Secretary  of  Defense  for  Personnel  and 
Readiness  accepted  changes  for  8  of  the  10  hospitals,  and  the  military 
services  made  further  modifications  to  the  recommendations.3 

Subsequent  to  DOD  initiating  the  Study,  Section  713  of  the  National 
Defense  Authorization  Act  (NDAA)  for  Fiscal  Year  2015  included  a 
provision  for  DOD  to  submit  the  Study  to  the  congressional  defense 
committees  and  also  listed  nine  elements,  some  with  multiple  items,  to  be 
incorporated  as  part  of  the  report.4  In  addition,  the  NDAA  included  a 
provision  for  us  to  review  the  Study  with  regard  to  its  methodology  and 


1  Figures  in  nominal  dollars.  We  determined  these  figures  through  analysis  of  Defense 
Health  Program  budget  documentation. 

department  of  Defense,  Report  on  Military  Health  System  Modernization:  Response  to 
Section  713  of  the  Carl  Levin  and  Howard  P.  “Buck”  McKeon  National  Defense 
Authorization  Act  for  Fiscal  Year  2015  (P.L.  1 13-291). 

3DOD  did  not  revise  its  cost  savings  estimate  based  on  these  changes  to  the  Study’s 
recommendations. 

4Pub.  L.  No.  113-291,  Section  713. 


Page  1 


GAO-16-820  Military  Health  Modernization 


the  data  used  to  support  it.  DOD  provided  the  Study  to  the  congressional 
defense  committees  in  February  201 6. 5  We  assessed  the  extent  to  which 
the  Study  (1)  addressed  each  of  the  nine  elements  included  in  the  NDAA; 
(2)  followed  an  approach  that  is  consistent  with  relevant  generally 
accepted  research  standards;  and  (3)  utilized  key  practices  for  cost 
savings  estimates. 

To  address  our  first  objective,  we  compared  Part  III  of  DOD’s  submission 
to  Congress,  the  data  appendix,  with  the  nine  elements  listed  in  the 
NDAA  for  Fiscal  Year  2015.  In  evaluating  the  data  appendix,  we 
considered  an  element  “addressed”  if  it  included  all  of  the  items  listed  in 
the  NDAA;  “partially  addressed”  if  it  included  some,  but  not  all,  of  the 
items;  and  “not  addressed”  if  it  did  not  include  any  of  the  items  within  the 
element.  We  discussed  the  data  appendix  and  our  preliminary 
observations  based  on  our  analyses  with  officials  from  the  Study  team 
within  the  Defense  Health  Agency. 

To  address  our  second  objective,  we  compared  the  Study  with  selected 
generally  accepted  research  standards  relevant  to  the  Study,  such  as  the 
need  for  clearly  stated  objectives,  reasonable  assumptions,  and  efforts  to 
assess  and  address  data  reliability.  These  standards  were  developed  by 
reviewing  research  literature  and  DOD  guidance  and  identifying 
frequently  occurring,  generally  accepted  research  standards  that  are 
relevant  for  defense  studies,  and  were  initially  used  in  a  prior  review  of  a 


5DOD’s  Report  on  Military  Health  System  Modernization  (the  Study)  to  Congress  includes 
three  parts.  Part  I  is  the  language  of  section  713  of  the  NDAA  for  Fiscal  Year  201 5;  Part  II 
is  DOD’s  Military  Health  System  Modernization  Study  Team  Report ;  Part  III  is  DOD’s 
detailed  responses,  including  data,  requested  in  Section  713.  For  purposes  of  this  report, 
we  refer  to  all  three  parts  collectively  as  the  Study. 
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DOD  study  on  mobility  capabilities.6  We  determined  that  these  standards 
are  still  current  and  relevant  for  the  purposes  of  this  report.  We  selected 
those  standards  that  were  most  relevant  to  the  Study,  in  particular  those 
standards  related  to  the  “design”  and  “execution”  of  a  defense  study.  For 
further  information  on  the  development  and  application  of  these  generally 
accepted  research  standards,  see  appendix  I.  We  also  interviewed 
officials  from  each  of  the  Study’s  working  groups  within  the  Defense 
Health  Agency  and  representatives  of  each  of  the  military  services 
concerning  their  analytical  and  decision-making  process.  To  assess 
DOD’s  efforts  to  address  data  reliability,  we  compared  the  Study  with 
selected  principles  from  the  standards  that,  among  other  things,  state  that 
it  is  good  practice  for  a  study  to  describe  how  researchers  determined  the 
data  used  were  valid  and  reliable  for  a  study’s  purposes  and  to  state  and 
adequately  explain  the  limitations,  if  any,  of  the  data  used  to  support  it.  To 
conduct  this  comparison,  we  (1)  reviewed  the  Study  for  passages  in 
which  data  reliability  is  discussed;  (2)  utilized  a  structured  data-collection 
instrument  to  query  Study  officials  regarding  their  efforts  to  assess  the 
reliability  of  data  used  in  the  Study;  and  (3)  systematically  searched  and 
reviewed  work  papers  provided  by  the  Study  team  to  identify  descriptions 
of  how  study-team  members  gathered  and  assessed  the  reliability  of  the 


6See  GAO,  Defense  Transportation:  Study  Limitations  Raise  Questions  about  the 
Adequacy  and  Completeness  of  the  Mobility  Capabilities  Study  and  Report,  GAO-06-938 
(Washington,  D.C.:  Sept.  20,  2006).  This  report  includes  a  list  of  questions  to  ask  in 
assessing  the  application  of  generally  accepted  research  standards.  To  develop  these 
standards,  we  reviewed  research  literature  and  DOD  guidance  and  identified  frequently 
occurring,  generally  accepted  research  standards  that  are  relevant  for  defense  studies 
that  define  a  quality  or  sound  and  complete  study.  The  following  were  our  sources  for 
these  standards,  with  the  citations  for  the  current  versions  of  these  documents:  GAO, 
Government  Auditing  Standards:  2011  Revision,  GAO-12-331G  (Washington,  D.C.: 
December  2011);  GAO,  Designing  Evaluations,  GAO-12-208G  (Washington,  D.C.: 
January  2012);  GAO,  Dimensions  of  Quality,  GAO/QTM-94-1  (Washington,  D.C.: 
February  2004);  RAND  Corporation,  RAND  Standards  for  High-Quality  Research  and 
Analysis,  CP-413  (5/15)  (Santa  Monica,  CA.:  May  2015);  Air  Force  Office  of  Aerospace 
Studies,  Analysts  Handbook:  On  Understanding  the  Nature  of  Analysis  (January  2000); 
Air  Force,  Office  of  Aerospace  Studies,  Air  Force  Analysis  Handbook,  A  Guide  for 
Performing  Analysis  Studies:  For  Analysis  of  Alternatives  or  Functional  Solution  Analysis 
(July  2004);  Department  of  Defense,  DOD  Modeling  and  Simulation  (M&S)  Verification, 
Validation,  Accreditation  (VV&A),  Instruction  5000.61  (Washington,  D.C.:  Dec.  9,  2009); 
Department  of  Defense,  Support  for  Strategic  Analysis,  Directive  8260.05  (Washington, 
D.C.:  July  7,  2011);  and  Department  of  Defense,  Implementation  of  Data  Collection, 
Development,  and  Management  for  Strategic  Analyses,  Instruction  8260.2  (Washington, 
D.C.:  Jan.  21,  2003).  We  determined  the  standards  were  applicable  for  evaluating  the 
Study. 
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data  used  in  their  analyses.  We  did  not  independently  assess  the 
reliability  of  the  source  data  that  DOD  used  in  developing  the  Study. 
Rather,  as  part  of  our  examination  of  the  Study’s  adherence  to  the 
relevant  generally  accepted  standards  for  defense  research,  we  asked 
DOD  officials  what  steps  they  took  to  assess  the  reliability  of  the  data 
used  in  developing  their  respective  sections  of  the  final  report,  and  we 
reviewed  supporting  documentation  provided  by  the  officials  to  identify 
any  known  data  reliability  issues. 

To  address  our  third  objective,  we  compared  the  2016  Study’s  cost 
savings  estimates  for  the  changes  to  10  military  hospitals  recommended 
in  the  Study  and  the  related  achievement  of  the  Study’s  goals  for 
transferring  health  care  to  DOD’s  own  facilities  and  increasing  active-duty 
provider  productivity  with  key  practices  we  derived  from  cost-estimating, 
budgeting,  and  related  guidance.  The  practices  call  for,  among  other 
things,  including  all  significant  costs  and  assessing  the  reliability  of  the 
data  used  to  develop  the  estimate.7  We  discussed  the  Study’s  estimate 
with  officials  from  the  Study  team  within  the  Defense  Health  Agency. 

To  address  all  of  our  objectives,  we  met  with  representatives  from 
relevant  organizations  across  DOD,  including  from  the  Office  of  the 
Assistant  Secretary  of  Defense  (Health  Affairs),  the  Defense  Health 
Agency,  and  representatives  of  the  medical  commands  and  agencies  of 
the  Army,  Navy,  and  Air  Force. 

We  conducted  this  performance  audit  from  October  2015  to  September 
2016  in  accordance  with  generally  accepted  government  auditing 


7We  reviewed  numerous  federal  guidance  documents  related  to  cost  estimating, 
accounting  standards,  economic  analysis,  and  budgeting  and  identified  broad  themes  that 
can  be  applied  to  evaluating  cost  analyses  and  estimates,  including  those  of  cost  savings. 
The  guidance  documents  we  reviewed  include:  GAO,  GAO  Cost  Estimating  and 
Assessment  Guide:  Best  Practices  for  Developing  and  Managing  Capital  Program  Costs , 
GAO-09-3SP  (Washington,  D.C.:  March  2009);  Office  of  Management  and  Budget 
Circular  No.  A-1 1 ,  Preparation,  Submission  and  Execution  of  the  Budget  (August  201 1 , 
superseded  by  an  August  2012  issuance);  Federal  Accounting  Standards  Advisory  Board, 
Statement  of  Federal  Financial  Accounting  Standards  4  (June  201 1 );  Department  of 
Defense  Instruction  7041.3,  Economic  Analysis  for  Decisionmaking  ( Nov.  7,  1995);  and 
Department  of  Defense  Financial  Management  Regulation  7000. 14-R,  Volume  4,  Chapter 
22,  Cost  Funding  (May  2010).  Although  each  of  these  documents  may  not  apply  to  these 
circumstances  as  a  legal  matter,  we  believe  that  they  collectively  contain  broad  themes 
that  can  be  applied  to  evaluating  cost  analyses. 
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standards.  Those  standards  require  that  we  plan  and  perform  the  audit  to 
obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  audit  objectives.  We  believe 
that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives. 


Background 


Overview  and  Structure  Of  The  MHS  is  a  complex  organization  that  provides  health  services  to 
the  MHS  almost  10  million  beneficiaries  across  a  range  of  care  venues,  including 

its  own  direct  care  system  in  hospitals  and  clinics,  through  its  purchased 
care  network  of  private  providers,  and  on  the  battlefield  in  support  of 
contingency  operations.  Responsibility  for  the  delivery  of  care  is  shared 
among  the  Office  of  the  Assistant  Secretary  of  Defense  (Health  Affairs), 
the  military  services,  and  the  Defense  Health  Agency.  The  Office  of  the 
Assistant  Secretary  of  Defense  (Health  Affairs)  reports  to  the  Under 
Secretary  of  Defense  for  Personnel  and  Readiness,  who  in  turn  reports  to 
the  Secretary  of  Defense.  The  Army  and  Navy  medical  commands  and 
the  Air  Force  medical  agency  report  through  their  service  chiefs  to  their 
respective  military  department  Secretaries  and  then  to  the  Secretary  of 
Defense.  The  Office  of  the  Assistant  Secretary  of  Defense  (Health  Affairs) 
manages  the  Defense  Health  Program  appropriation,  which  funds  the 
service  medical  departments.  However,  the  military  treatment  facilities, 
including  hospitals  and  clinics,  are  under  the  direction  and  control  of  the 
services,  which  maintain  the  responsibility  to  staff,  train,  and  equip  those 
commands  to  meet  mission  requirements.  The  MHS  collaboratively 
develops  strategy  to  meet  policy  directives  and  targets.  The  service 
components  and  the  Defense  Health  Agency  are  responsible  for 
execution.  Figure  1  depicts  the  MHS  organizational  structure. 
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Figure  1:  Military  Health  System  (MHS)  Organizational  Structure 


Policy  and  budget  oversight  provided  by  the  Under  Secretary  of  Defense  for  Personnel  and  Readiness  and  the  Assistant  Secretary  of  Defense  for  Health  Affairs 


Source:  Department  of  Defense  (DOD).  |  GAO-16-820 


Overview  of  the  Military 
Health  System 
Modernization  Study’s 
Findings  and 
Recommendations 


The  Study  was  chartered  by  the  Assistant  Secretary  of  Defense  (Health 
Affairs)  and  led  by  a  team  including  representatives  from  that  office,  as 
well  as  from  the  Defense  Health  Agency  and  from  the  medical  commands 
and  agencies  of  the  Army,  Navy,  and  Air  Force.  Officials  were  organized 
into  four  working  groups  that  collectively  made  up  the  Study  team.  Study 
officials  told  us  that  the  majority  of  the  Study’s  work  was  completed  in 
early  2013,  but  DOD  did  not  finalize  the  Study  until  early  2015,  and  it  did 
not  submit  the  Study  to  the  congressional  defense  committees  until 
February  2016.  The  Study  included  eight  recommendations  for  further 
study  and  analysis  in  a  wide  range  of  areas,  such  as  defining  medical 
force  readiness.  The  areas  addressed  in  the  Study  included  the  following: 

•  Medical  Readiness  Personnel  Requirements:  The  Study  team  sought 
to  determine  the  number  of  active-duty  uniformed  medical  personnel 
needed  to  support  military  operations. 

•  Health  Care  Demand:  The  Study  team  analyzed  the  demand  for 
health  care  in  geographical  medical  markets  both  within  DOD’s  direct 
care  system  of  hospitals  and  clinics  and  from  its  purchased  care 
network  of  private  providers.  The  Study  developed  goals  for  the 
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transfer  of  health  care  from  the  private  sector  to  DOD’s  own  facilities 
both  to  increase  the  efficiency  of  its  spending  by  making  better  use  of 
active-duty  providers  and  to  provide  sufficient  workload  for  them  to 
maintain  their  skills. 

•  Provider  Currency  Requirements:  The  Study  team  sought  to  develop 
a  metric  to  assess  the  currency  of  medical  personnel — that  is,  an 
assessment  of  the  maintenance  of  clinical  skills.  Currency,  in  turn, 
supports  the  readiness  of  medical  personnel,  meaning  their  ability  to 
provide  medical  support  during  operations.  However,  the  Study  team 
was  unable  to  agree  upon  a  clinical  currency  measure,  and  instead 
adopted  a  productivity  metric  as  a  proxy  for  currency.  The  productivity 
metric  aims  to  measure  active-duty  providers’  workload  level 
measured  in  Relative  Value  Units,  a  measure  of  health  care 
production.  The  accompanying  goal  is  for  active-duty  specialist 
providers  to  achieve  40  percent  of  the  median  productivity  of 
specialist  private-sector  providers  as  defined  by  a  major  medical 
survey.8 

•  Provider  Allocation  Model:  The  Study  team  developed  a  model  that 
matched  uniformed  active-duty  medical  providers  with  demand  for 
health  care  within  geographic  medical  markets,  with  the  assumption 
that  the  aforementioned  goals  for  the  transfer  of  health  care  from  the 
purchased  network  of  private  providers  to  DOD’s  facilities  and 
increasing  the  productivity  of  active-duty  providers  will  be  achieved. 
This  process  was  intended  to  demonstrate  over-  or  undersupply  of 
providers  in  certain  specialties.  The  analysis  found  that  for  some 
specialties,  there  was  insufficient  demand  or  other  constraining 
factors  that  prevented  the  matching  of  all  providers  to  markets.  For 
example,  based  on  the  Study’s  assumptions  regarding  the  transfer  of 
health  care  to  DOD  facilities  and  increasing  active-duty  provider 
productivity,  32  orthopedic  surgeons  could  not  be  matched  to  medical 
markets,  based  on  2013  data. 

•  Primary-Care  Analysis:  The  Study  examined  the  ratio  of  enrolled 
beneficiaries  to  primary-care  providers,  and  found  that  the  MHS  was 
close  to  its  goal  of  1 ,100  enrollees  per  primary-care  provider. 

•  Overseas  Hospital  Analysis:  Due  to  a  variety  of  cultural,  logistical,  and 
readiness  concerns,  the  Study  recommended  no  change  to  overseas 


8The  productivity  goal  is  based  on  data  from  the  2012  Medical  Group  Management 
Association  Survey. 
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military  hospitals.  Officials  told  us  that  this  was  the  primary  issue  of 
contention  within  DOD  due  to  concerns  surrounding  the  potential 
need  for  these  facilities  in  support  of  operations. 

The  Study  also  included  a  facility  analysis  of  24  small  hospitals  within  the 
MHS  to  determine  their  future  medical  capabilities,  such  as  whether  they 
should  continue  to  have  inpatient  facilities  or  birthing  centers.9  The  Study 
recommended  changes  for  10  of  the  24  hospitals,  including  closing 
inpatient  services  at  8  of  them.  The  Under  Secretary  of  Defense  for 
Personnel  and  Readiness  approved  proposed  changes  for  8  of  the  10 
facilities. 

The  Study  estimated  annual  net  cost  savings  of  $366  million  in  fiscal  year 
2012  dollars  for  the  recommended  changes  to  the  10  affected  hospitals 
and  for  achieving  its  goals  for  transferring  health  care  to  DOD  facilities 
and  increasing  active-duty  provider  productivity.  This  estimated  cost 
savings  included  about  $132  million  of  savings  associated  with  reductions 
in  civilian  Full  Time  Equivalents  (FTE),  $333  million  of  savings  associated 
with  decreased  purchased  care  costs,  and  $98  million  in  increased 
nonlabor  costs.10  Prior  to  the  release  of  DOD’s  report  to  Congress,  the 
Navy  completed  changes  to  two  of  the  hospitals  and  retained,  though 
reduced,  inpatient  services  at  the  third.  The  Army  also  approved  selected 
changes  at  three  of  its  facilities. 


9The  smallest  10  were  selected  before  the  development  of  the  Provider  Allocation  Model, 
but  officials  told  us  that  these  facilities  were  unlikely  to  have  any  providers  allocated  to 
them  under  the  model.  The  next  14  were  selected  based  on  the  results  of  the  model  and 
average  daily  patient  load  data. 

i  n 

All  figures  are  rounded. 
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DOD’s  Study 
Addressed  Four 
Elements,  Partially 
Addressed  Four 
Elements,  and  Did 
Not  Address  One 
Element  in  the  2015 
National  Defense 
Authorization  Act 


In  the  Report  on  Military  Health  System  Modernization  (the  Study),  DOD 
included  information  that  addressed  four  of  the  elements  listed  in  section 
713  of  the  NDAA  for  Fiscal  Year  2015,  partially  addressed  four  elements, 
and  did  not  address  one  element.11  For  example,  the  Study  addressed 
the  element  concerning  the  extent  to  which  DOD  consulted  with 
community  hospitals  to  determine  their  ability  to  absorb  additional 
patients  by  discussing  with  local  TRICARE  experts.  Similarly,  DOD 
addressed  the  element  concerning  the  recommendations  of  the  Secretary 
to  restructure  or  realign  military  medical  treatment  facilities  by  including  a 
list  of  recommended  changes  to  10  such  facilities. 

However,  the  Study  partially  addressed  four  elements  by  not  addressing 
every  item  of  the  relevant  element  or  including  incomplete  information. 

For  example,  the  Study  partially  addressed  the  element  concerning  the 
impact  of  recommended  changes  to  military  treatment  facilities  on  the 
access  of  eligible  beneficiaries  to  high-quality  health  care,  and 
satisfaction  with  such  care.  While  the  Study  listed  the  criteria  by  which 
DOD  evaluated  the  impact  of  recommended  changes  on  beneficiaries’ 
access  to  high-quality  health  care  and  satisfaction  with  such  care,  it  did 
not  provide  further  details  on  the  results  of  its  evaluation.  Similarly,  DOD 
partially  addressed  the  element  that  listed  14  items  concerning  hospitals 
considered  under  the  Study  from  2006  to  2012.  While  the  Study  provided 
information  addressing  11  of  these  items,  it  did  not  provide  full  information 
for  the  remaining  3,  such  as  the  omission  of  budgeted  funding  levels  for 
Navy  facilities  from  2006  through  2009.  The  Study  did  not  address  the 
element  concerning  the  impact  that  proposed  changes  to  military  medical 
treatment  facilities  would  have  on  timely  access  by  local  civilian 
populations  to  health  care  services.  While  the  Study  included  data  on 
civilian  hospital  capabilities,  it  did  not  explain  what  impact,  if  any,  the 
proposed  changes  would  have  on  access  to  such  facilities.  Appendix  II 
provides  further  information  on  our  analysis  of  the  extent  to  which  the 
Study  addressed  the  nine  elements  listed  in  Section  713  of  the  NDAA  for 
Fiscal  Year  2015. 


11  Pub.  L.  No.  113-291,  Section  713. 
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Six  of  the  nine  elements  required  a  single  response,  such  as  an 
explanation  or  analysis.  The  remaining  three  elements  listed  multiple 
items,  including  two  elements  that  required  information  concerning  all  of 
the  hospitals  considered  under  the  Study.  Although  DOD  assessed  24 
hospitals  in  its  Study,  it  chose  to  include  information  in  these  two 
elements  that  pertained  only  to  the  8  hospitals  for  which  the  Secretary  of 
Defense  approved  the  recommended  changes.  Neither  the  Study  nor  the 
appendix  provided  a  full  explanation  as  to  why  not  every  aspect  of  the 
required  items  was  included.  The  Study  appendix  noted  in  one  instance, 
for  the  item  concerning  mental-health  availability  at  training  centers,  that 
there  were  limited  data  available. 

According  to  senior  Study  team  officials,  DOD  compiled  the  data  required 
by  Section  713  after  the  Study  was  finalized  in  May  2015.  Further, 
officials  stated  that,  in  general,  the  compilation  of  these  data  did  not 
inform  the  Study’s  analysis,  findings,  or  recommendations,  because  DOD 
had  initiated  the  study  on  its  own  initiative  prior  to  the  enactment  of  the 
NDAA  for  Fiscal  Year  2015. 12  Although  Study  team  officials  stated  that 
they  independently  analyzed  some  of  the  same  data  elements  as  those 
listed  in  the  NDAA,  they  did  not  examine  all  such  elements.  However,  for 
the  data  elements  they  did  examine,  they  may  not  have  examined  the 
exact  attributes  specified  in  the  NDAA. 


DOD’s  Study  Did  Not 
Consistently  Follow 
Relevant  Generally 
Accepted  Research 
Standards 


Our  analysis  of  the  Study  found  that  it  followed  some,  but  not  all,  of  the 
relevant  generally  accepted  research  standards  for  research  design  and 
execution.13  Such  standards  contribute  to  a  sound  and  complete  study. 
According  to  these  standards,  a  study’s  design  should  include,  for 
example,  establishing  the  objectives,  scope,  and  methodology,  and 
identifying  study  assumptions.  Successful  execution  involves  ensuring 
that  the  methodology  is  carried  out  as  planned  and  ensuring  that  data 
used  in  the  study  are  sufficiently  valid  and  reliable  for  the  study’s 
purposes.  (See  app.  I  for  further  information  on  the  generally  accepted 
research  standards  that  we  used  to  conduct  our  assessment.) 
Specifically,  we  found  the  following: 


12Pub.  L.  No.  113-291,  §  713. 
13GAO-06-938. 
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•  The  Study’s  objectives,  scope,  and  methodology  were  generally 
consistent  with  relevant  accepted  research  standards. 

•  The  Study  did  not  sufficiently  mitigate  limitations  concerning  its 
analysis  of  the  required  number  of  active-duty  medical  and  civilian 
medical  personnel. 

•  The  Study  included  a  limited  assessment  of  clinical  currency — the 
requirements  necessary  to  maintain  the  skills  of  active-duty  medical 
providers. 

•  The  Study  established  goals  for  the  transfer  of  health  care  from 
DOD’s  purchased  care  network  into  its  own  network  of  hospitals  and 
clinics  and  for  increasing  the  productivity  of  active-duty  medical 
providers  without  a  strategy  explaining  how  these  goals  would  be 
achieved. 

•  The  Study’s  provider  allocation  model  did  not  reflect  differences 
among  the  military  services. 

•  The  Study  did  not  describe  steps  taken  to  ensure  the  reliability  of  the 
data  supporting  its  analyses  and  recommendations. 

In  general,  Study  officials,  including  those  from  the  various  working 
groups,  stated  that  they  were  limited  in  their  analysis  by  the  time  available 
to  them.  Study  officials  told  us  that  the  majority  of  the  Study’s  work  was 
completed  in  early  2013,  but  DOD  did  not  finalize  the  Study  until  early 
2015,  and  did  not  submit  it  to  the  congressional  defense  committees  until 
February  2016. 

The  Study’s  recommendations  for  future  work  and  analysis  position  DOD 
to  continue  important  efforts  to  modernize  the  MHS.  However,  without 
consistently  following  generally  accepted  research  standards,  DOD’s 
work  on  the  Study’s  recommendations  may  not  be  fully  sound  and 
complete.  By  addressing  the  areas  that  we  identified  in  our  review,  DOD 
can  strengthen  its  future  efforts. 
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Objectives,  Scope,  and 
Methodology  Were 
Generally  Consistent  with 
Relevant  Accepted 
Research  Standards 


The  Study’s  objectives,  scope,  and  methodology  were  generally 
consistent  with  relevant  accepted  research  standards  for  the  design  of  a 
study.  Generally  accepted  research  standards  state  that  objectives 
outline  what  a  study  is  intended  to  accomplish;  a  study’s  scope  defines 
the  subject  matter  that  will  be  assessed  and  reported  on,  and  is  directly 
tied  to  the  objectives;  and  a  study’s  methodology  should  be  consistent 
with  and  address  the  objectives.14 


The  Study  had  five  objectives  that  defined  the  subject  matter  to  be 
addressed,  and  all  of  these  related  to  DOD’s  high-level  goals  for  the 
Study  of  ensuring  the  readiness  of  the  medical  force  and  creating  a  more 
efficient  MHS.  The  Study’s  objectives  generally  corresponded  to  the  goals 
in  the  Study’s  Charter  document.  However,  two  goals  stated  in  the 
Study’s  Charter  were  not  addressed  in  these  objectives  or  in  the  Study’s 
execution.  Specifically,  the  Study  did  not  address  the  goals  to  finalize 
development  of  an  updated  joint  casualty  model,  which  would  provide  a 
cross-service  assessment  of  the  need  for  medical  support  in  support  of 
operations,  and  to  develop  recommended  civilian  medical  force  levels. 
(The  efforts  by  the  Study  team  to  address  the  Study’s  objectives  and 
Study’s  Charter  goals  are  discussed  later  in  this  report.)  Although  the 
Study  did  not  address  all  of  its  Charter’s  goals,  it  provided  new 
information  in  some  areas,  such  as  currency  of  active-duty  providers, 
which  DOD  can  use  to  develop  future  analysis. 

The  Study’s  scope  and  methodology  were  linked  with  the  objectives, 
consistent  with  generally  accepted  research  standards.  The  Study’s 
scope  included  all  medical  personnel  and  facilities  that  deliver  direct 
patient  care  in  the  Defense  Health  Program-funded  treatment  facilities. 
The  Study’s  methodology  involved  various  analyses  by  the  working 
groups  that  were  structured  around  each  of  the  five  objectives.  The 
Study’s  work  was  carried  out  by  corresponding  working  groups,  overseen 
by  an  overarching  group.  The  Study  also  provides  a  brief  discussion  of 
“risks  and  mitigation  strategies”  comprising  a  selected  list  of  six  changes 
in  the  environment  or  underlying  assumptions  that  could  affect  the 
Study’s  recommendations  and  an  accompanying  potential  alternative 
mitigation  strategy  to  address  each  change. 


14GAO-06-938. 
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The  Study  Did  Not  Fully 
Mitigate  Limitations 
Concerning  Its  Analysis  of 
the  Required  Number  of 
Active-Duty  and  Civilian 
Medical  Personnel 


The  Study  presented  data  on  the  size  and  composition  of  the  wartime 
active-duty  medical  force,  also  known  as  the  “operational  force.” 

However,  the  data  were  largely  drawn  from  service  models  with  known 
problems  and  not  based  on  new  analysis  conducted  by  the  Study  team. 
To  estimate  the  size  of  the  operational  force,  the  Study  presented  the 
sum  total  of  the  services’  individual  estimates  of  their  needs.  These 
estimates  are  produced  by  service-specific  models  and  planning  tools 
that  examine  a  common  set  of  defense  planning  scenarios.  However,  the 
Study  highlighted  differences  in  approach  between  the  services.  For 
example,  the  Army  chose  to  use  its  own  estimating  tool  because  the  Joint 
Medical  Planning  Tool  was  not  accredited  at  the  time  of  the  Study.  In 
contrast,  the  Navy  chose  to  use  an  early,  unaccredited  version  of  the 
Joint  Medical  Planning  Tool.  Similarly,  while  the  Army  used  the  Joint 
Integrated  Contingency  Model  to  develop  its  casualty  estimates,  which 
are  an  input  into  the  various  planning  tools,  the  Navy  used  four  separate 
models,  including  two  prototypes.  The  Air  Force  does  not  use  casualty 
estimates,  but  rather  an  estimate  of  the  number  of  likely  evacuees  from 
theater.  The  Study  noted  that  a  separate  independent  analysis  conducted 
by  the  Joint  Staff  correlated  with  the  services’  estimates  for  the 
operational  force.  In  addition,  Study  team  officials  stated  that  the  services’ 
processes  for  developing  their  “generating”  and  “institutional”  forces, 
which  include  personnel  responsible  for  issues  such  as  graduate  medical 
education  and  headquarters  staff  functions,  were  not  shared  with  the 
Study  team.  Previous  reviews  of  the  necessary  size  of  the  medical  force 
have  highlighted  the  differences  between  analyses  based  on  service¬ 
generated  estimates  and  analyses  based  on  DOD-wide  estimates. 
Specifically,  during  the  1990s,  two  analyses  based  on  DOD-wide 
estimates  found  the  services  had  excess  medical  personnel  relative  to 
their  needs,  while  one  analysis  based  on  service-generated  estimates 
found  the  services  needed  nearly  all  of  their  medical  personnel  to  meet 
their  needs.15 


Generally  accepted  research  standards  state  that  limitations  and 
constraints,  which  are  generally  found  in  studies,  should  be  identified  and 
reasonably  mitigated.16  While  the  Study  clearly  outlined  the  limitations  of 


15These  studies  arose  from  requirements  under  the  National  Defense  Authorization  Act 
(NDAA)  for  Fiscal  Year  1992. 

16GAO-06-938. 
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its  service-oriented  approach,  the  Study  did  not  fully  mitigate  them,  such 
as  by  explaining  how  these  limitations  affected  the  results  of  the  analysis. 
Officials  with  the  working  group  that  addressed  readiness  requirements 
stated  that  their  methodology  was  limited  by  the  time  available  for 
analysis.  Specifically,  the  officials  stated  that  they  knew  they  would  be 
unable,  in  the  time  allowed,  to  conduct  a  full,  independent  analysis  of  the 
number  of  active-duty  medical  personnel  necessary  to  support 
contingency  operations,  and  that  they  therefore  relied  upon  estimates 
from  the  military  services.  Officials  stated  that  it  would  be  feasible  to 
conduct  an  independent  analysis  of  the  required  number  of  active-duty 
medical  personnel  if  there  were  greater  information  sharing  from  the 
services  and  more  time  to  complete  such  an  analysis.  While  we  recognize 
the  time  constraints  faced  by  the  Study  team,  addressing  these  issues 
would  provide  more  meaningful  results  as  DOD  considers  further 
changes  to  the  MHS.  Absent  such  efforts  to  address  these  limitations, 
DOD  will  not  have  a  full  assessment  of  its  medical  workforce  needs. 

In  addition,  the  Study’s  Charter  included  a  goal  to  determine  civilian 
personnel  levels,  but  the  Study  did  not  include  such  an  assessment  in  its 
scope.  Civilians  represent  a  significant  proportion  of  the  MHS  workforce. 
For  example,  in  2015,  civilians  represented  61  percent  of  the  Army’s 
medical  positions.  Civilian  personnel  levels  also  have  a  direct  effect  on 
active-duty  personnel  levels,  and  DOD  policy  requires  the  services  and 
defense  agencies  to  determine  the  appropriate  mix  of  manpower  (military 
and  civilian)  and  private-sector  support.17  DOD’s  Medical  Readiness 
Review,  an  outgrowth  of  the  2006  Quadrennial  Defense  Review,  found 
that  efficiencies  could  be  made  by  converting  military  billets  to  civilian 
positions,  although  a  prohibition  on  such  conversions  was  later  put  in 
place.  Without  addressing  the  role  of  civilians  in  the  MHS,  the  Study’s 
analysis  does  not  present  a  complete  picture  of  DOD’s  needs. 


The  Study  Included  a 
Limited  Assessment  of 
Clinical  Currency 


The  Study  included  a  limited  assessment  of  clinical  currency  that  did  not 
sufficiently  identify  or  mitigate  several  limitations  and  constraints.  As 
noted  earlier,  the  Study  team  was  unable  to  agree  upon  a  measure  of  the 
clinical  currency  of  active-duty  medical  personnel — that  is,  an  assessment 


17Department  of  Defense  Instruction  1 1 00.22,  Policy  and  Procedures  for  Determining 
Workforce  Mix  (Apr.  12,  2010). 
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of  the  maintenance  of  skills.  Currency,  in  turn,  supports  the  readiness  of 
medical  personnel,  meaning  their  ability  to  provide  medical  support  during 
operations.  The  Study  stated  that  the  Study  team  attempted  to  identify  an 
industry  standard  for  clinical  currency,  but  none  existed.  In  addition,  a 
Study  official  stated  that  the  working  group  had  a  limited  time  frame  to 
complete  its  work.  The  Study  team  instead  adopted  a  productivity  metric 
as  a  proxy  for  currency,  along  with  an  accompanying  goal  for  active-duty 
specialist  providers  to  achieve  40  percent  of  the  median  productivity  of 
private-sector  specialist  providers  as  defined  by  a  major  medical  survey. 

However,  some  Study  team  officials  expressed  doubts  concerning  the 
accuracy  of  provider  productivity  data  used  to  develop  the  proxy  measure 
for  clinical  currency.  Specifically,  officials  stated  that  they  believe  that  the 
MHS  as  a  whole  does  not  accurately  record  Relative  Value  Units,18 
meaning  that  the  MHS  does  not  accurately  record  providers’  workload. 
Further,  a  senior  Study  team  official  stated  the  military  services  and 
individual  military  treatment  facilities  differ  in  how  they  record  this 
information,  and,  due  to  these  differences,  the  Study  team  adjusted  the 
Relative  Value  Unit  figures  for  a  number  of  hospitals,  sometimes  with  a 
margin  of  error  of  up  to  50  percent.  Generally  accepted  research 
standards  state  that  limitations  and  constraints  should  be  identified  and 
reasonably  mitigated.  However,  the  limitations  in  the  accuracy  of 
information  concerning  providers’  workload  were  not  identified  or 
mitigated  in  the  Study.  Having  accurate  workload  information  is  important 
to  establishing  a  sound  standard  for  maintaining  the  clinical  skills  of 
medical  providers. 

Further,  in  comparing  the  productivity  of  active-duty  medical  providers 
with  civilian  medical  providers,  the  Study  reduced  the  amount  of  time 
active-duty  providers  are  available  to  practice  medicine  because  of  the 
“readiness  cost”  of  time  that  they  must  devote  to  military-specific  duties 
and  requirements,  such  as  physical  training,  military  training,  and  federal 
holidays.  The  Study  team’s  approach  relied  on  subject-matter  experts  to 
estimate  the  time  active-duty  providers  had  available  to  practice 
medicine,  and  a  Study  official  said  there  were  no  reliable  data  concerning 
the  average  number  of  clinical  hours  worked  by  active-duty  providers. 


1 8 

Relative  Value  Units  are  a  metric  of  health-care  output  that  is  calculated  based  on  the 
type  and  complexity  of  an  episode  of  provision  of  care. 
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Other  approaches,  such  as  time  studies  of  active-duty  providers,  are 
more  rigorous,  but  the  Study  official  stated  that  the  working  group’s 
approach  was  limited  by  time  constraints. 

The  Study  contained  a  recommendation  to  further  study  the  costs  of 
medical  force  readiness,  and  DOD  is  currently  working  to  establish  a 
metric  for  medical  force  readiness.  However,  without  the  benefit  of  more 
rigorous  analysis  concerning  active-duty  providers’  workload  and  the  time 
they  must  devote  to  military-specific  duties  and  responsibilities,  DOD  will 
not  have  a  sound  and  accurate  standard  for  maintaining  the  clinical  skills 
of  medical  providers. 


The  Study  Established 
Goals  for  Transferring 
Health  Care  into  Its  Own 
Network  of  Hospitals  and 
Clinics  and  Increasing  the 
Productivity  of  Providers 
without  a  Strategy 
Explaining  How  These 
Goals  Would  Be  Achieved 


The  Study,  in  developing  its  findings  and  recommendations,  assumed 
that  DOD  would  be  successful  in  its  goals  for  transferring  health  care  from 
patients  being  treated  in  DOD’s  purchased  private-sector  network  into  its 
own  military  treatment  facilities,  as  well  as  in  achieving  related  goals  for 
improving  provider  productivity.  The  Study  recommended  the 
establishment  of  associated  goals,  and  DOD  adopted  such  goals  in 
March  2014.  However,  this  approach  was  not  consistent  with  the 
generally  accepted  research  standard  that  assumptions  should  be 
reasonable.  Specifically,  the  Study’s  assumptions  regarding  its  goals  for 
transfer  of  health  care  to  DOD’s  facilities  and  increasing  active-duty 
provider  productivity  are  unsupported  because  DOD  has  not  developed 
an  accompanying  strategy  describing  how  these  goals  can  be  achieved. 


Our  prior  work  has  identified  six  selected  leading  practices  of  effective 
federal  strategic  planning,  which  we  derived  in  part  from  the  Government 
Performance  and  Results  Act  (GPRA),  as  updated  by  the  GPRA 
Modernization  Act  of  2010,  associated  guidance,  and  our  prior  work.19 
Specifically,  these  leading  practices  are  to:  (1)  define  the  mission  and 
goals,  (2)  define  strategies  that  address  management  challenges  and 
identify  resources  needed  to  achieve  goals,  (3)  ensure  leadership 


1 Q 

For  example,  see  GAO,  Executive  Guide:  Effectively  Implementing  the  Government 
Performance  and  Results  Act,  GAO/GGD-96-1 18  (Washington,  D.C.:  June  1,  1996);  Tax 
Administration:  IRS  Needs  to  Further  Refine  Its  Tax  Filing  Season  Performance 
Measures,  GAO-03-143  (Washington,  D.C.:  Nov.  22,  2002);  and  Managing  for  Results: 
Strengthening  Regulatory  Agencies’  Performance  Management  Practices, 
GAO/GGD-00-10  (Washington,  D.C.:  Oct.  28,  1999). 


Page  16 


GAO-16-820  Military  Health  Modernization 


involvement  and  accountability,  (4)  involve  stakeholders,  (5)  coordinate 
with  other  federal  agencies,  and  (6)  develop  and  use  performance 
measures.  Study  team  officials  told  us  that,  notwithstanding  the  absence 
of  a  strategy,  transfer  of  health  care  to  DOD  facilities  is  a  priority  and  that 
such  goals  were  modest  and  achievable.  However,  the  Study  set  out 
goals  for  transferring  health  care  into  its  own  facilities  for  large  groups  of 
beneficiaries,  such  as  TRICARE  for  Life  beneficiaries,  for  whom,  unlike 
active-duty  military  and  their  families,  the  MHS  has  few  tools  to 
encourage  their  use  of  military  treatment  facilities. 

In  addition,  transfer  of  health  care  into  DOD  facilities  has  been  an  MHS 
goal  for  several  years,  but  limited  progress  has  been  made.  For  example, 
the  Congressional  Budget  Office  reported  in  201 420  that  most  of  the 
growth  in  outpatient  utilization  of  health  care  services  within  the  MHS  has 
occurred  within  the  purchased  care  system,  while  military  hospitals  and 
clinics  have  maintained  a  relatively  constant  workload  since  2005. 
Similarly,  it  is  uncertain  that  DOD  can  achieve  its  related  productivity 
goals  without  an  accurate  means  for  measuring  providers’  workload,  as 
discussed  earlier.  Without  a  strategy  that  incorporates  leading  practices,  it 
remains  unclear  whether  or  how  DOD  can  achieve  its  goals  to  transfer 
health  care  from  the  purchased  care  network  into  its  own  network  of 
facilities. 


The  Study’s  Provider 
Allocation  Model  Did  Not 
Reflect  Differences  among 
the  Military  Services 


The  Study  included  a  provider  allocation  model  that  identified  the  medical 
specialties  needed  with  anticipated  health  care  demand  within  specific 
geographic  medical  markets.  The  model’s  results  highlighted  specialties 
for  which  the  MHS  has  too  many  medical  specialists  relative  to 
anticipated  market  demand  for  various  health  care  services  and  other 
specialties  for  which  the  MHS  has  too  few  physicians  to  accommodate 
market  demand.  Generally  accepted  research  standards  state  that 
models  should,  among  other  things,  represent  a  reasonably  complete 
range  of  conditions  and  be  appropriate  for  the  intended  purpose.21 
However,  an  important  aspect  of  the  provider  model  did  not  reflect  the 
standards  in  that  it  unreasonably  assumed  that  medical  providers  from 


pA 

Congressional  Budget  Office,  Approaches  to  Reducing  Federal  Spending  on  Military 
Health  Care  (January  2014). 

21  GAO-06-938. 
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the  different  services  were  interchangeable.  Specifically,  the  Study’s 
model  for  allocation  of  specialist  physicians  based  on  anticipated  health 
care  demand  in  geographic  markets  did  not  account  for  the  military- 
service  ownership  of  the  relevant  hospitals  and  the  physicians’  military 
service.  Instead,  it  matched  the  number  of  physicians  across  all  services 
within  a  given  specialty,  such  as  orthopedic  surgery,  to  anticipated  health 
care  demand  in  geographic  markets,  regardless  of  the  service  ownership 
of  the  medical  treatment  facility  in  that  market.  While  such  arrangements 
do  exist  in  “multiservice  markets,”  such  as  the  National  Capital  Region 
and  San  Antonio,  they  are  less  common  elsewhere.  The  model  could 
potentially  provide  results  that  do  not  reflect  MHS  business  practices  in 
the  real  world.  For  example,  it  could  suggest  that  there  are  sufficient 
physicians  to  provide  a  given  type  of  specialty  care  to  a  particular 
geographic  medical  market,  when  in  reality  there  are  not  sufficient 
physicians  from  that  medical  market’s  primary  military  service  to  assign. 
Currently,  active-duty  providers  generally  practice  within  their  own 
services’  hospitals  in  a  team  context  unique  to  their  military  service.  The 
Study  emphasized  the  view  that  it  is  important  for  physicians  to  practice 
medicine  within  this  team  context  of  other  specialists  and  nurses  from  the 
same  service,  which  can  be  seen  within  the  services’  different 
configurations  of  medical  personnel  in  a  deployed  setting. 

CNA  independently  assessed  one  part  of  DOD’s  model.  However,  its 
assessment  included  only  the  mechanics  of  the  model  and  not  an 
examination  of  the  supporting  data  used  in  the  model.  The  Study  stated 
that  the  Study  team  relied  on  military-service  representatives  to  verify 
these  data.  In  addition,  CNA  noted  that  alternative  approaches  could  yield 
different  results  depending  on  assumptions  concerning  transfer  of  health 
care  to  DOD  facilities  and  increased  active-duty  provider  productivity.  The 
Study  asserted  that  its  modeling  results  were  reasonable  given  the  time 
constraints  of  the  Study.  The  Study  recommended  further  work  to  refine 
and  formalize  the  model  and  use  it  to  inform  decisions  concerning  the 
delivery  of  health  care  services.  However,  unless  DOD  addresses 
differences  between  the  services  and  the  effects  of  the  model’s  practical 
application,  the  model’s  usefulness  will  be  limited  and  could  lead  to 
impractical  decisions. 
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The  Study  Did  Not 
Describe  Steps  Taken  to 
Ensure  the  Reliability  of 
the  Data  Supporting  Its 
Analyses  and 
Recommendations 


The  Study  and  the  data  appendix  did  not  describe  steps  that  the  Study 
team  took  to  ensure  that  the  data  used  to  support  the  Study  and  data 
appendix  were  sufficiently  reliable  for  the  Study’s  purposes.  Generally 
accepted  research  standards  state  that  the  data  used  in  a  study  should 
be  valid  for  the  intended  research  purpose;  in  other  words  it  represents 
what  it  purports  to  represent.22  In  addition,  documenting  a  study’s 
analytical  and  decision-making  process  helps  to  ensure  the  accuracy, 
completeness,  and  credibility  of  study  results.  However,  we  found  that  the 
study  presented  limited  information  on  its  efforts  to  assess  data  reliability. 
For  example,  the  Study’s  data  appendix  did  not  document  the  source  of 
its  data  in  several  instances.  Specifically,  the  data  source  is  provided  for  9 
of  15  data  metrics,  but  not  for  the  other  6  metrics. 


Because  the  Study  presented  limited  information  on  the  Study  team’s 
efforts  to  assess  the  reliability  of  the  data  used,  we  provided  each  working 
group’s  lead  official  with  a  standard  set  of  questions  concerning  their 
efforts  to  ensure  data  reliability.  Specifically,  we  asked  the  working  group 
leads  which  data  sources  they  used  in  their  respective  analyses.  Their 
responses  indicated  that  they  used  five  databases  and  one  data  set  that 
were  not  mentioned  in  the  main  body  of  the  Study  or  the  data  appendix. 
We  questioned  the  working  group  leads  concerning  steps  taken  to  ensure 
data  reliability,  such  as  whether  they  consulted  with  the  source  or  owner 
of  the  data  to  determine  what  methods  were  used  to  ensure  reliability  or 
whether  they  took  other  steps  to  ensure  reliability,  such  as  conducting 
their  own  assessment  or  reliance  on  existing  assessments.  Working 
group  officials  stated  that  they  generally  relied  on  the  reliability  processes 
of  the  owners  of  the  data  to  ensure  the  data  provided  were  reliable.  For 
example,  two  of  the  working  groups  stated  that  they  used  databases  that 
are  commonly  used  within  the  MHS,  and  that  these  databases  are  subject 
to  oversight  reviews  and  periodic  validity  checks.  An  official  from  one 
working  group  stated  that  the  group  also  conducted  reviews  of  relevant 
data  to  check  for  completeness  of  records  related  to  inpatient  care  and 
found  missing  data.  The  working  group  developed  adjustment  factors  and 
applied  them  by  market  and  provider  specialty  to  impute  missing  data 
where  possible.  An  official  from  another  working  group  stated  that  the 
group  used  data  provided  by  contractors,  and  that  DOD’s  subject-matter 
experts  routinely  discuss  report  content  with  the  contractors  as  a  way  to 


22GAO-06-938. 
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verify  the  validity  and  reliability  of  the  data  provided  as  part  of  the 
performance  assessment  of  the  contract.  In  addition,  we  interviewed 
service  officials  regarding  the  accuracy  and  reliability  of  the  data  used  in 
the  Study,  and  officials  from  the  Navy  disputed  the  accuracy  of  the 
Study’s  reported  physician  productivity  data.  Army  officials  stated  that 
they  had  some  differences  in  data  interpretation  based  on  their  own 
analysis,  which  led  them  to  some  different  recommendations. 

When  asked  if  there  were  any  limitations  on  the  use  of  specific  data 
sources,  working  group  officials  reported  to  us  some  limitations  that  were 
not  specifically  described  in  the  main  body  of  the  Study  or  data  appendix. 
For  example,  one  working  group  official  mentioned  that  military  treatment 
facility  bed  counts  and  ambulatory  surgery  counts  were  not  consistently 
reported,  though  officials  stated  that  these  data  did  not  appear  to  have  a 
significant  effect  on  the  group’s  findings.  Another  limitation  concerned  the 
consistency  of  data  from  the  Medical  Expense  and  Performance 
Reporting  System  affecting  the  assessment  of  a  primary-care  provider’s 
availability.  An  official  noted  that  as  a  result  of  this  limitation,  the  final 
analysis  of  primary-care  capacity  was  curtailed  and  specific 
recommendations  were  limited. 

A  systematic  review  of  over  300  supporting  work  papers  provided  to  us  by 
Study  officials  found  multiple  examples  of  known  data  or  analysis 
limitations  that,  if  the  Study  was  adhering  to  generally  accepted  research 
standards,  would  have  been  conveyed  in  the  Study.  For  example,  one  of 
the  work  papers  noted  data  reliability  issues  with  certain  overseas-related 
data  which  were  not  noted  in  the  report.  One  work  paper  noted  some 
limitations  in  a  data  source  used  in  the  study  that  were  not  discussed  in 
the  report,  and  another  work  paper  mentioned  having  to  manually  adjust 
some  data  metrics,  which  was  not  noted  in  the  Study.  Conveying  data  or 
analysis  limitations  is  important  so  that  decision  makers  relying  on  the 
Study  can  fully  understand  the  quality  and  soundness  of  the  research. 
Without  sufficiently  demonstrating  the  accuracy,  reliability,  and  limitations 
of  the  data  upon  which  DOD  makes  policy  changes,  decision  makers  do 
not  have  assurance  that  the  Study’s  findings  and  recommendations  are 
based  on  sound  information. 
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DOD’s  Estimate  of 
Cost  Savings  from 
Changes  to  Small 
Hospitals  Did  Not 
Fully  Reflect  Key 
Practices 


DOD’s  estimated  cost  savings  for  the  recommended  changes  to  10  small 
hospitals  and  the  related  achievement  of  goals  related  to  transfer  of 
health  care  to  DOD  facilities  and  increased  active-duty  provider 
productivity  included  a  net  estimate  of  savings  and  costs,  but  did  not  fully 
reflect  key  practices  for  cost  and  cost  savings  estimates.  For  example, 
DOD  included  estimated  savings  from  three  categories — reductions  in 
civilian  personnel,  net  decreases  in  purchased  care,  and  net  increases  in 
nonlabor  costs — and  listed  general  sources  of  potential  savings. 

However,  the  Study  did  not  fully  reflect  key  practices  we  derived  from 
cost-estimating,  budgeting,  and  related  guidance.  These  key  practices 
emphasize  the  importance  of  an  appropriate  level  of  detailed 
documentation,  the  inclusion  of  all  significant  costs,  and  an  assessment 
of  the  reliability  of  the  data  used  to  develop  the  estimate,  among  others.23 

First,  the  Study  did  not  provide  an  appropriate  level  of  detail  concerning 
the  calculation  of  the  estimated  cost  savings  or  the  assumptions  upon 
which  they  were  based.  For  example,  the  Study’s  cost  savings  estimate 
did  not  include  details  concerning  the  number  of  military  personnel  who 
would  be  moved  from  the  10  affected  facilities  to  achieve  the  savings 
associated  from  transfer  of  health  care  to  DOD  facilities  and  increased 
active-duty  provider  productivity.  In  addition,  of  the  total  estimated 
savings,  $333  million  was  associated  with  changes  that  were  based  on  an 
assumption  concerning  DOD’s  ability  to  achieve  its  goals  related  to 
transfer  of  health  care  to  DOD  facilities  and  increased  active-duty 
provider  productivity  without  stating  a  strategy  to  reach  those  targets. 
Absent  a  clear  and  likely  achievable  strategy,  the  predicted  cost  savings 
associated  with  transfer  of  health  care  to  DOD  facilities  and  increasing 
active-duty  provider  productivity  are  questionable. 


po 

We  reviewed  numerous  federal  guidance  documents  related  to  cost  estimating, 
accounting  standards,  economic  analysis,  and  budgeting  and  identified  broad  themes  that 
we  believe  can  be  applied  to  evaluating  cost  analyses  and  estimates,  including  those  of 
cost  savings.  The  guidance  documents  we  reviewed  include  GAO-09-3SP;  Office  of 
Management  and  Budget  Circular  No.  A-1 1,  Preparation,  Submission  and  Execution  of 
the  Budget;  Federal  Accounting  Standards  Advisory  Board,  Statement  of  Federal 
Financial  Accounting  Standards  4;  Department  of  Defense  Instruction  7041.3,  Economic 
Analysis  for  Decisionmaking  (Nov.  7,  1995);  and  Department  of  Defense  Financial 
Management  Regulation  7000. 14-R,  Volume  4,  Chapter  22,  Cost  Funding.  Although  each 
of  these  documents  may  not  apply  to  these  circumstances  as  a  legal  matter,  we  believe 
that  they  collectively  contain  broad  themes  that  can  be  applied  to  evaluating  cost 
analyses. 
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Second,  the  Study’s  predicted  net  savings  estimate  did  not  include  all 
potentially  significant  costs.  Specifically,  its  cost  savings  estimate  did  not 
address  the  effect  of  the  following  costs: 

•  Costs  associated  with  a  reduction-in-force  of  civilian  FTEs:  The  Study 
estimated  savings  of  $132  million  associated  with  the  elimination  of 
1,324  civilian  FTEs  across  the  MHS  due  to  the  aforementioned 
assumption  of  greater  productivity  of  active-duty  providers.  However, 
we  have  previously  reported  that  DOD’s  efforts  to  reduce  civilian 
FTEs  do  not  always  result  in  planned  decreased  costs  of  a 
proportionate  magnitude  to  the  decrease  in  FTEs.  For  example,  in 
2016,  we  found  that  across  DOD,  from  fiscal  years  2012  through 
2016,  civilian  FTEs  declined  by  3.3  percent,  but  civilian  personnel 
costs  declined  by  only  0.9  percent,  adjusted  for  inflation.24 

•  Onetime  costs  to  realign  active-duty  providers  to  other  hospitals:  The 
Study  recommended  changes  that  would  realign  active-duty  providers 
to  other  facilities.  However,  movement  of  these  providers,  the 
associated  active-duty  medical  professionals  with  whom  they  work  in 
a  team  setting,  and  their  families  will  incur  costs  such  as  permanent 
change  of  station  expenditures.  Study  officials  told  us  that  they  did  not 
include  these  costs  because  permanent  change  of  station  costs  vary 
widely  and  final  decisions  concerning  where  personnel  would  be 
relocated  rest  with  the  military  services.  We  previously  reported  on  the 
costs  of  permanent  change  of  station  moves  and  found  that  these 
moves,  on  average,  cost  about  $6,700  per  servicemember  in  201 4. 25 

•  Construction  costs,  if  applicable,  at  facilities  losing  or  gaining  active- 
duty  providers:  The  Study  recommended  that  a  number  of  inpatient 
facilities  be  closed  or  converted  to  ambulatory  surgery  centers,  but 
our  analysis  found  that  the  Study  did  not  identify  estimated  costs 
associated  with  these  changes.  Construction  costs  also  may  be 
applicable  at  facilities  gaining  active-duty  providers,  but  Study  officials 
stated  that  execution  of  such  changes  would  be  the  responsibility  of 


24GAO,  Civilian  and  Contractor  Workforces:  Complete  Information  Needed  to  Assess 
DOD’s  Progress  for  Reductions  and  Associated  Savings,  GAO-16-172  (Washington,  D.C.: 
Dec.  23,  2015). 

25GAO,  Military  Compensation:  DOD  Needs  More  Complete  and  Consistent  Data  to 
Assess  the  Costs  and  Policies  of  Relocating  Personnel,  GAO-15-713  (Washington,  D.C.: 
Sept.  9,  2015). 


Page  22 


GAO-16-820  Military  Health  Modernization 


the  services.  However,  without  specifically  addressing  these  costs  or 
justifying  their  exclusion,  the  Study’s  cost  savings  estimate  leaves 
open  the  question  of  whether  potential  savings  could  be  reduced  by 
such  costs. 

Third,  DOD’s  savings  estimate  did  not  describe  the  steps  taken  by  the 
Study  team  to  assess  the  reliability  of  cost  data  used  to  develop  the  cost 
savings  estimate.  As  discussed  earlier  in  this  report,  the  Study  did  not  cite 
the  data  sources  from  which  the  cost  estimates  were  drawn,  and  did  not 
discuss  what  limitations,  if  any,  such  data  might  have  had  for  the 
purposes  to  which  they  were  applied. 

As  a  result  of  these  weaknesses,  DOD  may  not  have  accurately 
estimated  the  net  savings  that  might  result  from  its  proposed  hospital 
changes.  As  DOD  considers  additional  changes  to  its  network  of  military 
treatment  facilities,  decision  makers  do  not  have  a  full  and  accurate 
assessment  of  all  relevant  costs  and  savings.  Without  a  full  and  accurate 
assessment,  DOD  is  not  in  a  position  to  determine  whether  its  efforts  to 
respond  to  the  Study’s  recommendations  will  result  in  savings  or, 
alternatively,  in  additional  costs. 


Conclusions 


In  initiating  the  Study,  DOD  began  the  process  of  examining  difficult  and 
complex  problems  affecting  the  MHS.  The  Study’s  recommendations  for 
further  analysis  and  work  in  a  number  of  areas  position  DOD,  over  time, 
to  explore  these  areas  and  take  actions  to  improve  the  effectiveness  and 
efficiency  of  the  MHS.  However,  our  analysis  of  the  Study  highlighted 
shortcomings  that  merit  review  and  mitigation  before  DOD  takes  definitive 
action  on  these  recommendations.  Specifically,  without  addressing  the 
issues  discussed  in  our  report,  DOD’s  work  to  address  the  Study’s 
recommendations  will  continue  to  reflect  the  various  unsupported 
assumptions,  unmitigated  limitations  and  constraints,  and  questions 
concerning  the  reliability  of  data  used  to  develop  the  Study’s 
recommendations  that  we  have  highlighted.  Similarly,  as  DOD  considers 
further  changes  to  its  network  of  facilities  and  the  medical  services  they 
offer,  the  analysis  supporting  these  decisions  will  be  incomplete  without  a 
full  and  accurate  picture  of  the  true  cost  of  such  changes. 


Rprnmmpnrlptinnc;  fnr  To  fully  assess  the  size  and  composition  of  the  medical  force,  we 
r\fc!OUI  I II I  lfc!l  lUdllUl  lo  IUI  recommend  that  the  Secretary  of  Defense  direct  the  Assistant  Secretary 

Executive  Action  of  Defense  (Health  Affairs)  to  conduct  a  new  analysis  of  the  required 
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number  of  active-duty  and  civilian  medical  personnel  that  mitigates  known 
limitations. 

To  strengthen  ongoing  efforts  to  analyze  the  costs  of  medical  force 
readiness  and  establish  clinical  currency  standards,  we  recommend  that 
the  Secretary  of  Defense  direct  the  Assistant  Secretary  of  Defense 
(Health  Affairs)  to  take  steps  to  identify  and  mitigate  limitations  regarding 
the  standard  for  maintaining  providers’  clinical  skills,  including  improving 
the  accuracy  of  information  concerning  providers’  workload  and 
conducting  an  analytically  rigorous  calculation  of  active-duty  providers’ 
time  devoted  to  military-specific  responsibilities. 

To  help  achieve  DOD’s  goals  for  transferring  health  care  into  its  own 
facilities  and  increasing  the  productivity  of  active-duty  medical  providers, 
we  recommend  that  the  Secretary  of  Defense  direct  the  Assistant 
Secretary  of  Defense  (Health  Affairs)  to  develop  a  strategy  for  achieving 
these  goals  that  reflects  the  leading  practices  of  effective  federal  strategic 
planning. 

To  strengthen  ongoing  efforts  within  DOD  to  address  the  Study’s 
recommendations  to  use  the  provider  model  outputs  to  inform  execution 
of  health  care  delivery  and  to  refine  the  model  for  future  use,  we 
recommend  that  the  Secretary  of  Defense  direct  the  Assistant  Secretary 
of  Defense  (Health  Affairs)  to  modify  DOD’s  model  to  reflect  the  military 
service  of  the  physicians  and  military  treatment  facilities  included  in  the 
model. 

To  strengthen  any  future  assessments  of  additional  changes  to  DOD’s 
network  of  military  treatment  facilities,  we  recommend  that  the  Secretary 
of  Defense  direct  the  Assistant  Secretary  of  Defense  (Health  Affairs)  to 
take  the  following  two  actions: 

•  describe  steps  taken  to  assess  the  reliability  of  data  supporting  the 
assessment,  including,  at  a  minimum,  the  sources  of  data,  data 
limitations,  and  efforts  to  test  data  reliability;  and 

•  include  in  any  accompanying  cost  estimates  an  appropriate  level  of 
detail,  all  significant  costs,  and  an  assessment  of  the  reliability  of  the 
data  supporting  the  cost  estimate. 


Agency  Comments 


In  written  comments  on  a  draft  of  this  report,  DOD  concurred  with  our  six 
recommendations  to  improve  DOD’s  analysis  of  the  size,  readiness,  and 
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efficiency  of  the  medical  force.  DOD’s  comments  are  reprinted  in 
appendix  III. 


We  are  sending  copies  of  this  report  to  the  appropriate  congressional 
committees,  the  Secretary  of  Defense,  the  Deputy  Secretary  of  Defense, 
the  Under  Secretary  of  Defense  for  Personnel  and  Readiness,  the 
Assistant  Secretary  of  Defense  (Health  Affairs),  the  Defense  Health 
Agency  Director,  the  Surgeon  General  of  the  Air  Force,  the  Surgeon 
General  of  the  Army,  and  the  Surgeon  General  of  the  Navy.  In  addition, 
the  report  is  available  at  no  charge  on  the  GAO  website  at 
http://www.gao.gov. 

If  you  or  your  staff  have  any  questions  regarding  this  report,  please 
contact  me  at  (202)  512-3604  or  farrellb@gao.gov.  Contact  points  for  our 
Offices  of  Congressional  Relations  and  Public  Affairs  may  be  found  on 
the  last  page  of  this  report.  GAO  staff  who  made  key  contributions  to  this 
report  are  listed  in  appendix  IV. 


Brenda  S.  Farrell 
Director 

Defense  Capabilities  and  Management 
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List  of  Committees 


The  Honorable  John  S.  McCain 
Chairman 

The  Honorable  Jack  Reed 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Thad  Cochran 
Chairman 

The  Honorable  Richard  J.  Durbin 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Mac  Thornberry 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Rodney  Frelinghuysen 
Chairman 

The  Honorable  Peter  J.  Visclosky 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Appendix  I:  Selection  of  Generally  Accepted 
Research  Standards  Relevant  for  the  Military 
Health  System  Modernization  Study 


To  guide  our  review  of  the  Department  of  Defense’s  (DOD)  Report  on 
Military  Health  System  Modernization  (the  Study),  we  identified  relevant 
generally  accepted  research  standards  for  the  design,  execution,  and 
presentation  of  findings  that  define  a  sound  and  complete  study.  We 
developed  these  standards  by  reviewing  research  literature  and  DOD 
guidance  and  identifying  frequently  occurring,  generally  accepted 
research  standards  that  are  relevant  for  defense  studies.  We  initially  used 
these  standards  in  a  prior  GAO  review  of  a  DOD  study  on  mobility 
capabilities.1  That  report  includes  a  list  of  36  questions  to  ask  in 
assessing  the  application  of  generally  accepted  research  standards.  To 
develop  these  standards,  we  reviewed  research  literature  and  DOD 
guidance  and  identified  frequently  occurring,  generally  accepted  research 
standards  that  are  relevant  for  defense  studies  that  define  a  high-quality 
or  methodologically  sound  and  complete  study.  The  following  were  our 
sources  for  these  standards: 

•  GAO,  Government  Auditing  Standards:  2011  Revision, 

GAO-1 2-331 G  (Washington,  D.C.:  December  2011); 

•  GAO,  Designing  Evaluations,  GAO-1 2-208G  (Washington,  D.C.: 
January  2012); 

•  GAO,  Dimensions  of  Quality,  GAO/QTM-94-1  (Washington,  D.C.: 
February  2004); 

•  RAND  Corporation,  RAND  Standards  for  High-Quality  Research 
and  Analysis,  CP-413  (5/15)  (Santa  Monica,  CA.:  May  2015); 

•  Air  Force  Office  of  Aerospace  Studies,  Analysts  Handbook:  On 
Understanding  the  Nature  of  Analysis  (January  2000); 

•  Air  Force,  Office  of  Aerospace  Studies,  Air  Force  Analysis 
Handbook,  A  Guide  for  Performing  Analysis  Studies:  For  Analysis 
of  Alternatives  or  Functional  Solution  Analysis  (July  2004); 


^AO,  Defense  Transportation:  Study  Limitations  Raise  Questions  about  the  Adequacy 
and  Completeness  of  the  Mobility  Capabilities  Study  and  Report,  GAO-06-938 
(Washington,  D.C.:  Sept.  20,  2006).  That  report  includes  a  list  of  36  questions  to  ask  in 
assessing  the  application  of  generally  accepted  research  standards. 
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Appendix  I:  Selection  of  Generally  Accepted 
Research  Standards  Relevant  for  the  Military 
Health  System  Modernization  Study 


•  Department  of  Defense,  DOD  Modeling  and  Simulation  (M&S) 
Verification,  Validation,  Accreditation  (VV&A),  Instruction  5000.61 
(Washington,  D.C.:  Dec.  9,  2009); 

•  Department  of  Defense,  Support  for  Strategic  Analysis,  Directive 
8260.05  (Washington,  D.C.:  July  7,  201 1 ); 

•  Department  of  Defense,  Implementation  of  Data  Collection, 
Development,  and  Management  for  Strategic  Analyses,  Instruction 
8260.2  (Washington,  D.C.:  Jan.  21, 2003). 

We  determined  that  these  standards  are  still  current  and  relevant  for  the 
purposes  of  this  report. 

We  confined  our  assessment  to  the  standards  related  to  design  (11  of  14 
standards)  and  execution  (10  of  15  standards)  because  the  National 
Defense  Authorization  Act  for  Fiscal  Year  2015  mandated  we  assess  the 
methodology  and  data  used  in  the  Study.  Consequently,  we  did  not 
assess  DOD’s  report  against  the  generally  accepted  research  standards 
for  presentation  of  results  (7  standards).  Of  the  applicable  standards 
related  to  design  and  execution,  we  combined  and  slightly  revised  the 
standards  to  match  the  particular  analyses  used  in  the  study.  Some  of  the 
design  and  execution  standards  were  not  relevant  to  our  assessment.  For 
example,  the  standards  concerning  analysis  of  scenarios  and  threats  are 
specific  to  certain  types  of  defense  studies  and  did  not  relate  to  the  study 
that  was  the  subject  of  our  review.  We  compared  DOD’s  approach  to  the 
Study  with  these  selected  generally  accepted  research  standards. 
Accordingly,  we  technically  reviewed  the  content  of  DOD’s  report  against 
the  following  18  standards  that  were  relevant  to  our  review: 

1.  Design: 

a.  Objectives,  Scope  and  Methodology:  Objectives  outline  what  a 
study  is  intended  to  accomplish.  A  study’s  scope  defines  the 
subject  matter  that  will  be  assessed  and  reported  on,  and  the 
scope  is  directly  tied  to  the  audit  objectives.  A  study’s 
methodology  should  be  consistent  with  and  address  a  study’s 
objectives. 

•  Is  the  study’s  design  clear? 

•  Is  the  study’s  objective  clearly  stated? 

•  Is  the  study’s  scope  clearly  defined? 
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Appendix  I:  Selection  of  Generally  Accepted 
Research  Standards  Relevant  for  the  Military 
Health  System  Modernization  Study 


•  Do  the  study  scope,  methodology,  and  objectives  fully 
address  the  study  charter  and  associated  guidance? 

•  Was  the  study  plan  followed? 

•  Were  deviations  from  the  study  plan  explained  and 
documented? 

b.  Assumptions:  Assumptions  should  be  reasonable,  consistent, 
appropriately  varied,  and  explicitly  identified  in  order  to  understand 
the  context  of  the  study’s  results.  Assumptions  define  the 
parameters  for  what  will  be  included  and  how  the  study  is 
executed. 

•  Are  the  assumptions  explicitly  identified? 

•  Are  the  assumptions  reasonable  and  consistent? 

•  Are  the  assumptions  varied  to  allow  for  sensitivity  analyses? 

c.  Limitations  and  Constraints:  Limitations  and  constraints,  which  are 
generally  found  in  studies,  should  be  identified  and  reasonably 
mitigated.  Research  standards  do  not  call  for  the  elimination  of  all 
limitations,  but  do  call  for  them  to  be  identified,  assessed, 
mitigated,  and  explained. 

•  Are  the  limitations  explicitly  identified? 

•  Are  the  major  constraints  identified  and  discussed? 

2.  Execution: 

a.  Models:  Models  should  be  reasonable  ones  to  consider,  represent 
a  reasonably  complete  range  of  conditions,  as  well  as  be 
appropriate  for  the  intended  purpose. 

•  Are  the  scenarios  that  were  modeled  reasonable  ones  to 
consider? 

•  Do  the  scenarios  represent  a  reasonably  complete  range  of 
conditions? 

•  Were  the  models  used  to  support  the  analyses  appropriate  for 
their  intended  purpose?  Have  they  been  described  and 
documented  adequately? 

•  Were  any  modeling  limitations  identified,  explained,  and 
justified? 
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Appendix  I:  Selection  of  Generally  Accepted 
Research  Standards  Relevant  for  the  Military 
Health  System  Modernization  Study 


b.  Data  reliability:  The  data  used  in  a  study  should  be  valid  for  the 
intended  research  purpose;  in  other  words,  they  represent  what 
they  purport  to  represent.  Documenting  a  study’s  analytical  and 
decision-making  process  helps  to  ensure  the  accuracy, 
completeness,  and  credibility  of  study  results. 

•  Were  the  data  used  valid  for  the  study’s  purposes? 

•  Were  the  data  used  sufficiently  reliable  for  the  study’s 
purposes? 

•  Were  any  data  limitations  identified  and  was  the  effect  of  the 
limitations  adequately  explained? 
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Appendix  II:  Comparison  of  DOD’s  Military 
Health  System  Modernization  Study  with 
Requirements  Listed  in  2015  National 
Defense  Authorization  Act 


Table  1  summarizes  our  assessment  of  the  extent  to  which  the 
Department  of  Defense  (DOD)  in  its  Report  on  Military  Health  System 
Modernization  (the  Study)  addressed  each  of  the  required  nine  elements 
listed  in  Section  713  of  the  National  Defense  Authorization  Act  (NDAA)  for 
Fiscal  Year  201 5.  Some  of  the  elements  required  that  multiple  items  be 
addressed.  We  considered  an  element  “addressed”  if  it  included  all  of  the 
items  listed  in  the  NDAA;  “partially  addressed”  if  it  included  some,  but  not 
all,  of  the  items;  and  “not  addressed”  if  it  did  not  include  any  of  the  items. 


Table  1:  Comparison  of  DOD’s  Military  Health  System  Modernization  Study  with  Elements  Listed  in  Section  713  of  the  NDAA 
for  Fiscal  Year  2015 

National  Defense  Authorization  Act  (NDAA)  elements  and  items 

Our  assessment 

(A)  During  the  period  from  2006  to  2012,  for  each  military  medical  treatment 
facility  considered  under  the  modernization  study — 

Partially  Addressed 

(i)  the  average  daily  inpatient  census 

Addressed 

(ii)  the  average  inpatient  capacity 

Partially  addressed:  Only  two  years  of  data  included. 

(iii)  the  top  five  inpatient  admission  diagnoses 

Addressed 

(iv)  each  medical  specialty  available 

Addressed 

(v)  the  average  daily  percent  of  staffing  available  for  each  medical 
specialty 

Addressed 

(vi)  the  beneficiary  population  within  the  catchment  area 

Addressed 

(vii)  the  budgeted  funding  level 

Partially  addressed:  Did  not  provide  budgeting 
funding  level  details  for  the  years  2006  through  2009 
for  Navy  hospitals.  Data  were  only  included  for  three 
Army  Community  Hospitals.  No  explanation  is 
provided  for  this  omission. 

(viii)  whether  the  facility  has  a  helipad  capable  of  receiving  medical 
evacuation  airlift  patients  arriving  on  the  primary  evacuation  aircraft 
platform  for  the  military  installation  served 

Addressed 

(ix)  a  determination  of  whether  the  civilian  hospital  system  in  which  the 
facility  resides  is  a  federally  designated  underserved  medical  community 
and  the  effect  on  such  community  from  any  reduction  in  staff  or  functions 
or  downgrade  of  the  facility 

Addressed 

(x)  If  the  facility  serves  a  training  center — 

(I)  a  determination  of  the  risk  with  respect  to  high-tempo,  live-fire 
military  operations,  treating  battlefield-like  injuries,  and  the  potential 
for  a  mass-casualty  event  if  the  facility  is  downgraded  to  a  clinic  or 
reduced  in  personnel  or  capabilities;  and 

(II)  a  description  of  the  extent  to  which  the  Secretary,  in  making  such 
determination,  consulted  with  the  appropriate  training  directorate, 
training  and  doctrine  command,  and  forces  command  of  each  military 
department; 

(I)  Partially  addressed:  Did  not  state  whether  facilities 
were  training  centers  or  explicitly  speak  to  battlefield 
type  injuries.  However,  did  assess  risk  to  operational 
mission  for  three  unspecified  “small”  military  treatment 
facilities. 

(II)  Partially  addressed:  Consultations  with  numerous 
service  stakeholders  are  noted,  but  not  specifically 
with  the  organizations  listed. 

(xi)  a  site  assessment  by  TRICARE  to  assess  the  network  capabilities  of 
TRICARE  providers  in  the  local  area; 

Addressed 
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Appendix  II:  Comparison  of  DOD’s  Military 

Health  System  Modernization  Study  with 

Requirements  Listed  in  2015  National  Defense 

Authorization  Act 

National  Defense  Authorization  Act  (NDAA)  elements  and  items 

Our  assessment 

(xii)  the  inpatient  mental  health  care  availability;  and 

Partially  addressed:  Only  limited  data  were  available. 

(xiii)  the  average  annual  inpatient  care  directed  to  civilian  medical 
facilities. 

Addressed 

(B)  For  each  military  medical  treatment  facility  considered  under  the 
modernization  study — 

Partially  Addressed 

(i)  the  civilian  capacity  by  medical  specialty  in  each  catchment  area; 

Addressed 

(ii)  the  distance  in  miles  to  the  nearest  civilian  emergency  care 
department; 

Addressed 

(iii)  the  distance  in  miles  to  the  closest  civilian  inpatient  hospital,  listed  by 
level  of  care  and  whether  the  facility  is  designated  a  sole  community 
hospital; 

Addressed 

(iv)  the  availability  of  ambulance  service  on  the  military  installation  and 
the  distance  in  miles  to  the  nearest  civilian  ambulance  service,  including 
the  average  response  time  to  the  military  installation; 

Addressed 

(v)  an  estimate  of  the  cost  to  restructure  or  realign  the  military  medical 
treatment  facility,  including  with  respect  to  bed  closures  and  the  civilian 
personnel  reduction;  and 

Partially  addressed:  This  figure  was  not  explicitly 
stated,  but  it  was  included  in  the  savings  estimates. 

(vi)  if  the  military  medical  treatment  facility  is  restructured  or  realigned,  an 
estimate  of — 

(1)  the  number  of  civilian  personnel  reduction,  listed  by  series; 

(1)  Partially  addressed:  A  net  figure  was  included,  but 
not  for  each  medical  treatment  facility  or  listed  by 
series. 

(II)  the  number  of  local  support  contracts  terminated;  and 

(III)  the  increased  cost  of  purchased  care. 

(II)  Not  addressed:  Study  stated  estimates  would  be 
made  during  implementation. 

(III)  Partially  addressed:  A  net  figure  was  included,  but 
not  for  each  medical  treatment  facility. 

(C)  The  results  of  the  modernization  study  with  respect  to  the 
recommendations  of  the  Secretary  to  restructure  or  realign  military  medical 
treatment  facilities 

Addressed 

The  Study  presented  its  findings  and 
recommendations  for  modernization  of  the  military 
healthcare  system. 

(D)  An  assessment  of  the  analysis  made  by  the  Secretary  to  inform  decisions 
regarding  the  modernization  of  the  military  health  care  system  in  the 
modernization  study 

Addressed 

The  Study  documented  the  analyses  used  to  inform 
the  recommendations  for  modernization. 

(E)  An  assessment  of  the  extent  to  which  the  Secretary  evaluated  in  the 
modernization  study  the  impact  on  the  access  of  eligible  beneficiaries  to 
quality  health 

Partially  Addressed 

(i)  changes  in  military  medical  treatment  facility  infrastructure; 

Partially  addressed:  The  Study  states  that  the  effect 
of  potential  changes  would  be  considered  during 
implementation  of  recommendations;  however,  quality 
and  satisfaction  not  addressed  in  the  report. 

(ii)  changes  in  staffing  levels  of  professionals;  and 

Partially  addressed:  The  Study  states  that  the  effect 
of  potential  changes  would  be  considered  during 
implementation  of  recommendations;  however,  quality 
and  satisfaction  not  addressed  in  the  report. 
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Appendix  II:  Comparison  of  DOD’s  Military 

Health  System  Modernization  Study  with 

Requirements  Listed  in  2015  National  Defense 

Authorization  Act 

National  Defense  Authorization  Act  (NDAA)  elements  and  items 

Our  assessment 

(iii)  changes  in  inpatient,  ambulatory  surgery,  and  specialty  care  capacity 
and  capabilities. 

Partially  addressed:  The  Study  listed  the  criteria  by 
which  DOD  evaluated  the  impact  of  recommended 
changes  on  beneficiaries’  access  to  high-quality 
health  care  and  satisfaction  with  such  care;  however, 
while  institutional  quality  data  could  relate  to  the 
request  for  impact  assessment,  it  remains  unclear 
whether  DOD  explicitly  used  the  data  in  this  manner. 

(F)  An  assessment  of  the  extent  to  which  the  Secretary  evaluated  in  the 
modernization  study  how  any  reduced  inpatient,  ambulatory  surgery,  or 
specialty  care  capacity  and  capabilities  at  military  medical  treatment  facilities 
covered  by  the  study  would  impact  timely  access  to  care  for  eligible 
beneficiaries  at  local  civilian  community  hospitals  within  reasonable  driving 
distances  of  the  catchment  areas  of  such  facilities 

Addressed 

The  Study  included  data  that  demonstrated  an 
analysis  of  the  effects  of  potential  changes  in  the 
hospitals  on  timely  access  to  care  for  beneficiaries, 
and  potential  for  treatment  at  nearby  civilian  hospitals. 

(G)  An  assessment  of  the  extent  to  which  the  Secretary  consulted  in 
conducting  the  modernization  study  with  community  hospitals  in  locations 
covered  by  the  study  to  determine  their  capacities  for  additional  inpatient  and 
ambulatory  surgery  patients  and  their  capabilities  to  meet  additional  demands 
for  specialty  care  services 

Addressed 

The  Study  noted  consultations  with  local  TRICARE 
experts  to  determine  the  capacity  of  community 
hospitals  to  absorb  additional  patients. 

(H)  An  assessment  of  the  extent  to  which  the  Secretary  considered  in  the 
modernization  study  the  impact  that  the  change  in  the  structure  or  alignment 
of  military  medical  treatment  facilities  covered  by  the  study  would  have  on 
timely  access  by  local  civilian  populations  to  inpatient,  ambulatory  surgery,  or 
specialty  care  services  if  additional  eligible  beneficiaries  also  sought  access  to 
such  services  from  the  same  providers 

Not  Addressed 

The  Study  included  data  on  civilian  hospital 
capabilities;  however,  it  did  not  explain  the  analysis  of 
these  data  that  led  the  Study  team  to  its  conclusions 
concerning  access  to  care. 

(1)  An  assessment  of  the  impact  of  the  elimination  of  health  care  services  at 
military  medical  treatment  facilities  covered  by  the  modernization  study  on 
civilians  employed  at  such  facilities 

Partially  Addressed 

The  Study  noted  the  number  of  civilian  full-time 
equivalents  that  will  be  downsized,  but  did  not 
otherwise  address  the  effect  of  these  changes  on 
civilian  personnel. 

Source:  GAO  analysis  of  DOD  data.  |  GAO-16-820 
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Appendix  III:  Comments  from  the 
Department  of  Defense 


THE  ASSISTANT  SECRETARY  OF  DEFENSE 

1200  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-1200 

SEP  0  6  20(5 

Ms.  Brenda  S.  Farrell 

Director,  Defense  Capabilities  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  NW 
Washington,  DC  20548 

Dear  Ms.  Farrell, 

Thank  you  for  the  opportunity  to  review  the  GAO  Draft  Report  GAO- 16-820, 
“DEFENSE  HEALTH  CARE  REFORM:  DOD  Needs  Further  Analysis  of  the  Size,  Readiness, 
and  Efficiency  of  the  Medical  Force,”  dated  August  8,  2016  (GAO  Code  100345).  Attached  is 
the  Department  of  Defense’s  response  to  this  report.  My  point  of  contact  is  Dr.  David  Smith.  Dr. 


HEALTH  AFFAIRS 


Smith  may  be  reached  at  (703)  681-8477,  or  at  david.j. smith  1 52.civ@mail.mil. 


(Jk 

\  S'Av.  f  >/ 


Karen  S.  Guice,  M.D.,  M.PtP. 
Acting 


Attachment: 
As  stated 
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GAO  DRAFT  REPORT  DATED  AUGUST  8,  2016 
GAO- 16-820  (GAO  CODE  100345) 

“DEFENSE  HEALTH  CARE  REFORM:  DOD  NEEDS  FURTHER  ANALYSIS  OF  THE 
SIZE,  READINESS,  AND  EFFICIENCY  OF  THE  MEDICAL  FORCE” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATION 


RECOMMENDATION  1:  To  fully  assess  the  size  and  composition  of  the  medical  force,  GAO 
recommends  that  the  Secretary  of  Defense  direct  the  Assistant  Secretary  of  Defense  (Health 
Affairs)  to  conduct  a  new  analysis  of  the  required  number  of  active  duty  and  civilian  medical 
personnel  that  mitigates  known  limitations. 

DoD  RESPONSE:  Concur  with  comment.  The  Military  Services  routinely  develop  the 
uniformed  military  requirements  using  models  and  processes  that  reflect  specific  Service 
missions  in  the  joint  force,  as  recognized  in  the  GAO  report.  The  Department  is  developing 
Essential  Medical  Capabilities  that  will  further  refine  the  development  of  uniformed 
requirements.  As  the  Department  continues  to  refine  its  assessment  of  its  medical  needs,  the 
civilian  workforce  will  be  assessed  against  cost  and  military  force  readiness  support 
requirements  to  develop  the  most  effective  force  mix. 

RECOMMENDATION  2:  To  strengthen  ongoing  efforts  to  analyze  the  costs  of  medical  force 
readiness  and  establish  clinical  currency  standards,  GAO  recommends  that  the  Secretary  of 
Defense  direct  the  Assistant  Secretary  of  Defense  (Health  Affairs)  to  take  steps  to  identify  and 
mitigate  limitations  regarding  the  standard  for  maintaining  providers’  clinical  skills,  including 
improving  the  accuracy  of  information  concerning  providers’  workload  and  conducting  an 
analytically  rigorous  calculation  of  active  duty  providers’  time  devoted  to  military-specific 
responsibilities. 

DoD  RESPONSE:  Concur  with  comment.  The  Department  is  developing  Essential  Medical 
Capabilities  as  well  as  appropriate  metrics  for  reporting  medical  force  readiness.  This  will  allow 
us  to  refine  our  assessment  of  provider  and  healthcare  team  skills  sustainment  requirements. 

This  effort  will  provide  the  foundation  for  routine  and  comprehensive  assessment  of  the 
capabilities  of  its  medical  forces  and  aligns  with  similar  reporting  on  the  balance  of  the 
Department’s  capabilities. 

RECOMMENDATION  3:  To  help  achieve  DoD’s  goals  for  transferring  health  care  into  its 
own  facilities  and  increasing  the  productivity  of  active  duty  medical  providers,  GAO 
recommends  that  the  Secretary  of  Defense  direct  the  Assistant  Secretary  of  Defense  (Health 
Affairs)  to  develop  a  strategy  for  achieving  these  goals  that  reflects  the  leading  practices  of 
effective  federal  strategic  planning. 

DoD  RESPONSE:  Concur.  The  Military  Health  System  is  a  continuous  learning  organization 
and  the  Assistant  Secretary  of  Defense  (Health  Affairs)  and  Service  Surgeons  General  are 
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committed  to  applying  the  leading  practices  to  improve  productivity  and  to  recapture  care  to  our 
military  treatment  facilities.  The  Military  Health  System  has  developed  a  strategic  focus  on 
primary  care  as  a  methodology  to  increase  recapture  as  well  as  provider  productivity.  The 
Military  Health  System  is  also  implementing  metrics  to  measure  both  recapture  as  well  as 
provider  productivity.  Results  from  these  efforts  will  be  reviewed  and  used  to  inform  the 
Departments  efforts  to  maintain  its  medical  forces. 

RECOMMENDATION  4:  To  strengthen  ongoing  efforts  within  DoD  to  address  the  Study’s 
recommendations  to  use  the  provider  model  outputs  to  inform  execution  of  health  care  delivery 
and  to  refine  the  model  for  future  use,  GAO  recommends  that  the  Secretary  of  Defense  direct  the 
Assistant  Secretary  of  Defense  (Health  Affairs)  to  modify  its  model  to  reflect  the  military  service 
of  the  physicians  and  military  treatment  facilities  included  in  the  model. 

DoD  RESPONSE:  Concur.  In  the  Department’s  largest  medical  markets,  we  have  implemented 
a  management  model  that  provides  flexibility  to  match  staff  capabilities  to  beneficiary  demand. 
Assessing  medical  facilities  in  these  markets  exclusively  through  a  Service  lens  would 
compromise  the  ability  to  fully  address  demand.  As  the  Department  continues  to  refine  its 
methodologies  for  assessing  it  medical  requirements  we  will  reflect  Service  affiliations.  The 
Provider  Allocation  Model  does  include  the  ability  to  identify  Service  affiliation. 

RECOMMENDATION  5:  To  strengthen  any  future  assessments  of  additional  changes  to 
DoD’s  network  of  military  treatment  facilities,  GAO  recommends  that  the  Secretary  of  Defense 
direct  the  Assistant  Secretary  of  Defense  (Health  Affairs)  to  describe  steps  taken  to  assess  the 
reliability  of  data  supporting  the  assessment,  including,  at  a  minimum,  the  sources  of  data,  data 
limitations,  and  efforts  to  test  data  reliability. 

DoD  RESPONSE:  Concur.  The  Department  continuously  assesses  the  adequacy  and 
completeness  of  its  medical  data.  We  continuously  evaluate  the  consistency  of  analyses  using 
basic  healthcare  data  to  ensure  that  the  data  is  complete  and  accurate.  This  resulted,  in  part,  in 
the  mitigations  highlighted  in  the  GAO  Report,  but  were  not  included  in  the  Modernization 
Study  report.  The  Department  will  continue  to  assess  the  appropriateness  of  detailed  data 
reliability,  limitations,  and  mitigations  in  future  analyses. 

RECOMMENDATION  6:  To  strengthen  any  future  assessments  of  additional  changes  to 
DoD’s  network  of  military  treatment  facilities,  GAO  recommends  that  the  Secretary  of  Defense 
direct  the  Assistant  Secretary  of  Defense  (Health  Affairs)  to  include  in  any  accompanying  costs 
estimates  an  appropriate  level  of  detail,  all  significant  costs,  and  an  assessment  of  the  reliability 
of  the  data  supporting  the  cost  estimate. 

DoD  RESPONSE:  Concur.  As  the  Department  continues  to  refine  its  methodologies  for 
assessing  medical  requirements  they  will  include  reasonable  and  assessable  costs  in  our  estimates 
of  financial  impacts. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and  investigative 
arm  of  Congress,  exists  to  support  Congress  in  meeting  its  constitutional 
responsibilities  and  to  help  improve  the  performance  and  accountability  of  the 
federal  government  for  the  American  people.  GAO  examines  the  use  of  public 
funds;  evaluates  federal  programs  and  policies;  and  provides  analyses, 
recommendations,  and  other  assistance  to  help  Congress  make  informed 
oversight,  policy,  and  funding  decisions.  GAO’s  commitment  to  good  government 
is  reflected  in  its  core  values  of  accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost  is 
through  GAO’s  website  (http://www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  website  newly  released  reports,  testimony,  and  correspondence.  To 
have  GAO  e-mail  you  a  list  of  newly  posted  products,  go  to  http://www.gao.gov 
and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of  production  and 
distribution  and  depends  on  the  number  of  pages  in  the  publication  and  whether 
the  publication  is  printed  in  color  or  black  and  white.  Pricing  and  ordering 
information  is  posted  on  GAO’s  website,  http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card,  MasterCard, 

Visa,  check,  or  money  order.  Call  for  additional  information. 

Connect  with  GAO 

Connect  with  GAO  on  Facebook,  Flickr,  Twitter,  and  YouTube. 

Subscribe  to  our  RSS  Feeds  or  E-mail  Updates.  Listen  to  our  Podcasts. 

Visit  GAO  on  the  web  at  www.gao.gov. 

To  Report  Fraud, 

Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Website:  http://www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Katherine  Siggerud,  Managing  Director,  siggerudk@gao.gov,  (202)  512-4400, 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125, 
Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngd@gao.gov,  (202)  512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 

Washington,  DC  20548 

Strategic  Planning  and 
External  Liaison 

James-Christian  Blockwood,  Managing  Director,  spel@gao.gov,  (202)  512-4707 
U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7814, 
Washington,  DC  20548 
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